Likelihood and Fairness in Multidimensional Item Response Theory

In multidimensional item response theory, it is possible for the estimate of a subject’s ability in some dimension to decrease after they have answered a question correctly. When this occurs, the fairness of the estimate may be called into question. This paper investigates how and when this form of non-monotonicity occurs. We demonstrate that many response models and statistical estimates can produce nonmonotone results and one popular class is guaranteed to do so. In light of these findings, the appropriateness of statistical inference for assigning scores in high-stakes testing is called into question.